System for controlling vehicle display based on occupant&#39;s gaze

ABSTRACT

A system for controlling a vehicle display based on an occupant&#39;s gaze includes: a first occupant gaze recognition unit to recognize a gaze of a first occupant in a vehicle; a second occupant gaze recognition unit to recognize a gaze of a second occupant in the vehicle; an interest information display area crossover confirmation and priority determination unit to determine interest information from the gaze of the first occupant and the gaze of the second occupant, to confirm whether areas for displaying interest information on a windshield are mutually crossover with each other, and to determine a priority for display; and a display control unit to control a display in accordance with the priority for display.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to and the benefit of Korean PatentApplication No. 10-2021-0137536, filed on Oct. 15, 2021, Korean PatentApplication No. 10-2021-0137537, filed on Oct. 15, 2021, and KoreanPatent Application No. 10-2021-0137543, filed on Oct. 15, 2021, thedisclosures of which are incorporated herein by reference in theirentirety.

BACKGROUND 1. Technical Field

The present disclosure relates to a system for recognizing gazeinformation of a plurality of occupants in a vehicle, and moreparticularly, to a system for controlling a vehicle display when avehicle windshield display area regarding external interest informationis crossover.

2. Related Art

An occupant in a vehicle may want to stare at an interesting object(e.g., tree, building, another vehicle, bicycle, motorcycle, etc.) amongvarious objects viewed through a window of the vehicle while driving,and to check information about the interest object.

The related art describes a configuration for synthesizing augmentedreality content with a forward object that a driver is highly likely tobe watching and outputting the synthesized reality content.

According to the related art, a configuration for minimizing the gazemovement of the driver has been disclosed, but there has been a problemthat it is difficult to meet various needs related to the externalobject identification of the occupant.

SUMMARY

This Summary is provided to introduce a selection of concepts insimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

The present disclosure is proposed to solve the aforementioned problem,and the present disclosure is directed to a system for controlling avehicle display according to a preset standard by recognizing gazeinformation of an occupant in a vehicle and displaying information aboutexternal interest objects of a plurality of occupants on a vehiclewindshield display when the external interest objects different fromeach other.

In addition, the present disclosure is directed to a system fordetermining a display area for external interest information using gazeinformation of a plurality of occupants in a vehicle, and controlling avehicle display according to the characteristics of the externalinterest information.

In addition, the present disclosure is directed to a system forcontrolling a vehicle display area position for external interestinformation of a plurality of occupants in a vehicle and controlling adisplay time for the external interest information.

In one general aspect, a system for controlling a vehicle display basedon an occupant's gaze includes: a first occupant gaze recognition unitconfigured to recognize a gaze of a first occupant in a vehicle; asecond occupant gaze recognition unit configured to recognize a gaze ofa second occupant in the vehicle; an interest information display areacrossover confirmation and priority determination unit configured todetermine interest information from the gaze of the first occupant andthe gaze of the second occupant, to determine whether areas fordisplaying the interest information in a windshield are mutuallycrossover with each other, and to determine a priority of a display; anda display control unit configured to control a display in accordancewith the priority of the display.

The display control unit may be configured to control the display inconsideration of preset initiative information when a display area forthe interest information of the first occupant and a display area forthe interest information of the second occupant overlap with each other.

The display control unit may be configured to control the display sothat the display area for the interest information of the first occupantis displayed on a top position while covering the display area for theinterest information of the second occupant, when the first occupant isset to have an initiative over the second occupant.

The display control unit may be configured to control the display bymoving the display area for the interest information of the secondoccupant so that the display area for the interest information of thesecond occupant does not overlap with the display area for the interestinformation of the first occupant, when the first occupant is set tohave an initiative control over the second occupant.

The display control unit may be configured to control the display inconsideration of an importance of the interest information, when thedisplay area for the interest information of the first occupant and thedisplay area for the interest information of the second occupant overlapwith each other.

The display control unit may be configured to control the display todisplay the interest information of the first occupant, withoutdisplaying the interest information of the second occupant, when it isdetermined that the interest information of the first occupant hashigher importance than the interest information of the second occupant.

The display control unit may be configured to determine whether todisplay the interest information of the second occupant in the displayarea assigned to the first occupant, in consideration of at least one oftypes of content displayed in the display area assigned to the firstoccupant and an operation situation in relation to the display areaassigned to the first occupant.

The system may include a first occupant's external interest informationconfirmation unit configured to confirm external interest information ofthe first occupant from the gaze of the first occupant; a secondoccupant's external interest information confirmation unit configured toconfirm external interest information of the second occupant from thegaze of the second occupant; and an external interest informationdisplay area confirmation unit configured to confirm a display area forthe external interest information of the first occupant and a displayarea for the external interest information of the second occupant. Theinterest information display area crossover confirmation and prioritydetermination unit may be configured to determine the priority fordisplay, in consideration of recognition influence on the display areafor the external interest information of the first occupant by thedisplay area for the external interest information of the secondoccupant with relatively low importance of information display.

The display control unit may be configured to perform display control todisplay only the external interest information of the first occupantwithout displaying the external interest information of the secondoccupant when it is determined that the recognition influence exceeds apreset value.

The display control unit may be configured to perform control to displayoutline information in respect to an original position of an externalinterest object of the second occupant.

The display control unit may be configured to restore the display areafor the external interest information of the second occupant to an areabefore moving in consideration of the gaze of the first occupant.

In another general aspect, a method for controlling a vehicle displaybased on an occupant's gaze includes: (a) recognizing gaze informationof a plurality of occupants in a vehicle; (b) confirming interestinformation using the gaze information; (c) confirming whether displayareas for the interest information displayed on a windshield that is atransparent display crossover with each other; and (d) performingdisplay control when the display areas of the interest informationcrossover with each other.

In (d), the display control may be performed in consideration of presetinitiative control information according to a result of confirming thata display area for interest information of a first occupant and adisplay area for interest information of a second occupant overlap witheach other.

In (d), when the first occupant is set to have an initiative controlthan the second occupant, the display control may be performed so thatthe display area for the interest information of the first occupant isdisplayed at a top level while covering the display area for theinterest information of the second occupant.

In (d), when the first occupant is set to have an initiative controlthan the second occupant, the display control may be performed so thatthe display area for the interest information of the second occupant ismoved so as to not overlap with the display area for the interestinformation of the first occupant.

In (d), when the display area for the interest information of the firstoccupant and the display area for the interest information of the secondoccupant overlap with each other, the display control may be performedin consideration of importance of the interest information.

In (d), when it is determined that the interest information of the firstoccupant has higher importance than the interest information of thesecond occupant, the display control may be performed so that theinterest information of the second occupant is not displayed and theinterest information of the first occupant is displayed.

In (d), it may be determined whether the interest information of thesecond occupant is displayed in a display area assigned to the firstoccupant, in consideration of at least any one of types of contentdisplayed in the display area assigned to the first occupant and anoperation situation associated with the display area assigned to thefirst occupant.

According to the present disclosure, it is possible to obtain an effectof controlling a vehicle windshield display for each area according to apreset priority, in consideration of the needs of a plurality ofoccupants who want to check information related to different externalinterest objects through the vehicle windshield display.

In addition, according to the present disclosure, it is possible toobtain an effect of controlling a vehicle display according to a presetpriority, in consideration of a display area within the vehiclewindshield display for external interest information in which aplurality of occupants are interested.

In addition, according to the present disclosure, it is possible toobtain an effect of controlling a display duration time and a displayarea of the external interest information according to a presetpriority, in consideration of the needs of a plurality of occupants whowish to check information related to different external interest objectsthrough the vehicle windshield display.

However, the effects of the present disclosure are not limited to thosedescribed above, and other effects which are not mentioned will beclearly understood by those skilled in the art from the followingdescription.

Other features and aspects will be apparent from the following detaileddescription, the drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a view showing a system for controlling a vehicle based on anoccupant's intent according to an embodiment of the present disclosure.

FIGS. 2, 3, and 4 are views showing an indoor layout change based on anoccupant's intent according to the embodiment of the present disclosure.

FIG. 5 is a view showing a method for controlling a vehicle based on anoccupant's intent according to an embodiment of the present disclosure.

FIG. 6 is a view showing a vehicle display control system based on anoccupant's gaze according to another embodiment of the presentdisclosure.

FIG. 7A and FIG. 7B are views showing vehicle windshield display controlfor each area, in consideration of priority, according to anotherembodiment of the present disclosure.

FIG. 8A and FIG. 8B are views showing the vehicle windshield displaycontrol for each area, in consideration of priority during a relativemovement of an external interest object, according to another embodimentof the present disclosure.

FIG. 9 is a view showing a vehicle display control method based on anoccupant's gaze according to another embodiment of the presentdisclosure.

FIG. 10 is a view showing an external interest information display areadetermination and vehicle display control system based on an occupant'sgaze according to another embodiment of the present disclosure.

FIG. 11A, FIG. 11B, and FIG. 11C are views showing external interestinformation display area determination and display control according toanother embodiment of the present disclosure.

FIG. 12 is a view showing an external interest information display areadetermination and vehicle display control method based on an occupant'sgaze according to another embodiment of the present disclosure.

FIG. 13 is a view showing an occupant gaze departure determination andvehicle display control system according to another embodiment of thepresent disclosure.

FIGS. 14, 15, 16, and 17 are views showing an example of vehicle displaycontrol in an occupant gaze departure situation according to anotherembodiment of the present disclosure.

FIG. 18 is a view showing an example of vehicle display control upondeparture from control authority according to another embodiment of thepresent disclosure.

FIG. 19 is a view showing an occupant gaze departure determination andvehicle display control method according to another embodiment of thepresent disclosure.

FIG. 20 is a view showing a button display control in an occupantgaze-based vehicle display and initiative control system according toanother embodiment of the present disclosure.

FIGS. 21, 22, and 23 are views showing button display control accordingto another embodiment of the present disclosure.

FIG. 24 is a view showing an initiative switch in another occupant'sparticipation according to another embodiment of the present disclosure.

FIG. 25 is a view showing display control based on screen area divisionand gaze information upon another occupant's participation according toanother embodiment of the present disclosure.

FIG. 26 is a view showing a method for button display control in anoccupant gaze-based vehicle display and initiative control according toanother embodiment of the present disclosure.

FIG. 27 is a view showing a vehicle display control system fortransmitting external interest information according to anotherembodiment of the present disclosure.

FIG. 28A and FIG. 28B are views showing external interest informationdisplay and external interest information display area control accordingto another embodiment of the present disclosure.

FIG. 29 is a view showing a vehicle display control method for conveyingexternal interest information according to another embodiment of thepresent disclosure.

FIG. 30 is a view showing a vehicle display control system forsynthesizing and displaying external interest information according toanother embodiment of the present disclosure.

FIG. 31A, FIG. 31B, and FIG. 32 are views are views showing externalinterest information display and rotation according to still anotherembodiment of the present disclosure.

FIG. 33 is a view showing a vehicle display control method forsynthesizing and displaying external interest information according toanother embodiment of the present disclosure.

FIG. 34 is a view showing a vehicle display control system for storingand sharing external interest information according to anotherembodiment of the present disclosure.

FIG. 35A, FIG. 35B, and FIG. 35C are views showing a process ofcapturing the external interest information according to anotherembodiment of the present disclosure.

FIG. 36A and FIG. 36B are views showing a process of sharing theexternal interest information with a smartphone according to anotherembodiment of the present disclosure.

FIG. 37A, FIG. 37B, and FIG. 37C are views showing a process of storingand confirming external interest information for each occupant through atransparent display sliding according to another embodiment of thepresent disclosure.

FIG. 38A, FIG. 38B, and FIG. 38C are views showing the sharing of theexternal interest information between displays according to anotherembodiment of the present disclosure.

FIG. 39 is a view showing a vehicle display control method for storingand sharing external interest information according to anotherembodiment of the present disclosure.

FIG. 40 is a view showing a vehicle display area control system forexternal interest information according to another embodiment of thepresent disclosure.

FIG. 41A, FIG. 41B, and FIG. 41C are views showing a process ofcontrolling a vehicle display area for the external interest informationaccording to still another embodiment of the present disclosure.

FIG. 42 is a view showing a method of controlling a vehicle display areafor external interest information according to another embodiment of thepresent disclosure.

FIG. 43 is a view showing a vehicle display control system that displaysand stores the external interest information according to still anotherembodiment of the present disclosure.

FIG. 44A and FIG. 44B are views showing an indication of the externalinterest information according to still another embodiment of thepresent disclosure.

FIG. 45 is a view showing a vehicle display control method fordisplaying and storing the external interest information according tostill another embodiment of the present disclosure.

DETAILED DESCRIPTION

The aforementioned purposes of the present disclosure and otherpurposes, advantages and features, and how to achieve them, will becomeclear with reference to embodiments described below in detail with theaccompanying drawings.

However, the present disclosure is not limited to the embodimentsdescribed below but may be implemented in various forms different fromone another. The following described embodiments are only provided tofacilitate the purpose, configuration and effectiveness of the inventionto a person of ordinary skill in the art to which the present inventionbelongs. Thus, the scope of the present disclosure is defined by thedescription of the claims.

On the other hand, the terms used herein are intended to describeembodiments and are not intended to limit the present disclosure. Asused herein, the singular form also includes a plural form unlessotherwise noted in the phrase. The terms “comprises” and/or “comprising”as used in the specification do not preclude the presence or addition ofone or more other constituent elements, steps, operations and/orelements to the mentioned constituent elements, steps, operations and/orelements.

FIG. 1 is a view showing a system for controlling a vehicle based on anoccupant's intent according to an embodiment of the present disclosure.

The system for controlling a vehicle based on an occupant's intentaccording to the embodiment of the present disclosure includes: anoccupant gaze recognition unit 101 configured to recognize a gaze of anoccupant in a vehicle; a point of interest determination unit 103configured to determine a point of interest, among external objectsviewed through a side window of the vehicle in consideration of the gazeof the occupant; and a seat position control unit 105 configured tochange a seat position of the occupant to enable a continuous stare atthe point of interest as the vehicle moves.

The system for controlling a vehicle based on an occupant's intentaccording to the present disclosure further includes: an occupantgesture recognition unit 102 configured to recognize at least any one ofa touch input signal of the occupant and an air gesture of the occupantthat is applied to the side window of the vehicle configured as atransparent display; and the point of interest determination unit 103determines the point of interest as at least any one of the touch inputsignal and the air gesture is recognized.

The seat position control unit 105 transmits a control signal to rotatea seat in which the occupant is seated, in consideration of a travelingdirection and travel speed of the vehicle.

The seat position control unit 105 transmits a control signal to rotatea seat of a first occupant, in consideration of gazes of a plurality ofvehicle occupants that stare the same point of interest, and transmits acontrol signal to rotate a seat of a second occupant with forward orbackward sliding for the seat of the second occupant.

The system for controlling a vehicle based on an occupant's intentaccording to the present disclosure further includes a display controlunit 104 configured to control, when it is confirmed that the gazes ofthe plurality of vehicle occupants stare at the same point of interestand then it is determined that a field of view of the second occupant isblocked by the first occupant, an outer area, viewed through a firstside window close to a seating position of the first occupant, to bedisplayed through a second side window close to a seating position ofthe second occupant. At this time, the seat position control unit 105transmits the control signal to rotate the seat of the first occupant ina first direction, and transmits the control signal to rotate the seatof the second occupant in an opposite direction to the first direction.

FIGS. 2 to 4 are a view showing an indoor layout change based on anoccupant's intent according to the embodiment of the present disclosure.

Referring to FIGS. 2 to 4 , it is assumed that among persons in thevehicle traveling forward, there is a first occupant in a left rear seatand a second occupant in a right rear seat.

The first occupant is a person close to a transparent display of a leftside window, and performs touch operations on the transparent display,and the second occupant is a person far from the transparent display ofthe left side window, and performs an operation through an air gesture.

According to the embodiment of the present disclosure, the gaze of theoccupant is recognized and AR information personalized for each occupantbased on a gaze information is displayed through the transparent displayof the left side window.

Referring to FIG. 2 , when the first occupant inputs a touch signal intothe area of an external object (isolated spire) viewed through thetransparent display of the left side window, the point of interestdetermination unit 103 determines that an external particular object isthe interest object, in consideration of the gaze of the first occupantand the touch signal application area of the first occupant.

The display control unit 104 displays specific information about theinterest object through the transparent display of the left side window,in the form of text, images, icons, etc.

In addition, the display control unit 104 may provide the informationabout the interest object as a sound source through a speaker and mayalso independently provide the sound source to the first occupantthrough a directional speaker or a headrest speaker disposed at the leftrear seat.

As the vehicle travels forward, the external interest object viewedthrough the left side window is moved in a left backward directionrelative to a vehicle body.

When it is checked that the gaze of the first occupant stays on theinterest object and the first occupant turns his/her head in the leftbackward direction and intends to continuously stare at the interestobject, the seat position control unit 105 rotates the left rear seat,on which the first occupant is seated, in a counterclockwise direction,allowing the first occupant to continuously stare at the interest objectin a more comfortable position.

At this time, the seat position control unit 105 determines that thefirst occupant wants to continuously stare at the interest object when astaring duration time of the first occupant for the interest objectobtained through the occupant gaze recognition unit 101 is equal to orlonger than a preset time, and the seat position control unit 105controls a rotational speed of the left rear seat on which the firstoccupant is seated in accordance with the driving speed of the drivingvehicle and the relative movement speed of the external interest object.

When the seat position control unit 105 predicts that a sharp seatrotation will be required and will cause discomfort to the firstoccupant in a case in which the seat is rotated in accordance with therelative movement of the external interest object during high-speeddriving equal to or higher than a predetermine speed (e.g., 50 Km/h),the seat position control unit 105 does not perform the rotationoperation on the seat.

The occurrence of discomfort is predicted based on a predeterminedreference seat rotation speed.

At this time, the seat position control unit 105 determines whether toperform a seat rotation operation based not only on the driving speed ofthe driving vehicle, but also on a distance information between theexternal interest object and the vehicle.

For example, in a situation where an external interest object is 50meters away from the driving vehicle, when the seat is rotated so thatthe first occupant can continuously stare at the external interestobject at the driving speed, a sharp seat rotation is required, whereaswhen an external interest object is 500 meters away from the drivingvehicle, even if the seat is rotated so that the first occupant cancontinuously stare at the external interest object at the driving speed,a sharp seat rotation is not required.

Thus, the seat position control unit 105 comprehensively considers thedriving speed of the driving vehicle and the distance between theexternal interest object at which the first occupant wants tocontinuously stare and the driving vehicle, to determine whether torotate the seat of the first occupant so that the first occupant cancontinuously and comfortably stare at the external interest objectwithout turning his/her head.

As the vehicle travels further forward, when it is determined that theinterest object is out of the range of vision of the first occupant, theseat position control unit 105 rotates the left rear seat, on which thefirst occupant is seated, in a clockwise direction to return the leftrear seat to its original position.

In addition, the seat position control unit 105 rotates the left rearseat, on which the first occupant is seated, in the clockwise directionto return the left rear seat to its original position when the gazeinformation obtained through the occupant gaze recognition unit 101 nolonger stays on the interest object for a certain period of time orlonger.

Referring to FIG. 3 , it is assumed that the first occupant inputs atouch signal into a corresponding area while staring at an externalobject (isolated spire) viewed through the transparent display of theleft side window, and the second occupant performs a predetermined airgesture while staring at the same external object (isolated spire)viewed through the transparent display of the left side window.

That is, the first occupant and the second occupant designate the sameexternal object as an external point of interest (POI).

The point of interest determination unit 103 and determines that anexternal particular object is the interest object, in consideration ofthe gaze of the first occupant, the touch signal application area of thefirst occupant, the gaze of the second occupant, and the air gesture ofthe second occupant.

The display control unit 104 displays a specific information of theinterest object through the transparent display of the left side window,in the form of text, images, icons, etc.

In addition, the display control unit 104 may provide the informationabout the interest object as a sound source through a speaker, and inthis case, the display control unit 104 may also provide the soundsource to the first and second occupants, through a directional speakerfacing the occupants seated on the rear seats, a headrest speakerdisposed at the left rear seat, and a headrest speaker disposed at theright rear seat.

As the vehicle travels forward, the external interest object viewedthrough the left side window is moved in the left backward directionrelative to the vehicle body.

When it is confirmed that the gazes of the first occupant and the secondoccupant stay on the interest object and the first occupant and thesecond occupant are then continuously staring at the interest object byturning their heads in the left backward direction, the seat positioncontrol unit 105 rotates the left rear seat, on which the first occupantis seated, in the counterclockwise direction, allowing the firstoccupant to continuously stare at the interest object in a morecomfortable position.

In addition, in consideration of the field of view of the secondoccupant obscured by the first occupant as the external interest objectis relatively moved, the seat position control unit 105 slides the rightrear seat, on which the second occupant is seated, backward of thevehicle body and rotates the right rear seat in the counterclockwisedirection, so that the second occupant can more easily and continuouslystare at the interest object.

As the vehicle travels further forward, when it is determined that theinterest object is out of the ranges of visions of the first occupantand the second occupant, the seat position control unit 105 rotates theleft rear seat, on which the first occupant is seated, in the clockwisedirection, and slides forward the right rear seat on which the secondoccupant is seated and rotates the right rear seat in the clockwisedirection, to return the left and right rear seats to their originalseat positions.

Referring to FIG. 4 , it is assumed that the first occupant inputs atouch signal into the area while staring at the external object(isolated spire) viewed through the transparent display of the left sidewindow, and the second occupant performs a predetermined air gesturewhile staring at the same external object (isolated spire) viewedthrough the transparent display of the left side window.

That is, the first occupant and the second occupant designate the sameexternal object as the external POI.

The point of interest determination unit 103 determines that an externalparticular object is the interest object, in consideration of the gazeof the first occupant, the touch signal application area of the firstoccupant, the gaze of the second occupant, and the air gesture of thesecond occupant.

The display control unit 104 displays the specific information of theinterest object through the transparent display of the left side window,in the form of text, images, icons, etc.

In addition, the display control unit 104 may provide the informationabout the interest object as a sound source through a speaker, and inthis case, the display control unit 104 may also provide the soundsource to the first and second occupants, through a directional speakerfacing the occupants seated on the rear seats, a headrest speakerdisposed at the left rear seat, and a headrest speaker disposed at theright rear seat.

As the vehicle travels forward, the external interest object shownthrough the left side window is moved in a left backward directionrelative to the vehicle body.

When it is confirmed that the gaze of the first occupant stays on theinterest object and the first occupant is then continuously staring atthe interest object by turning his/her head in the left backwarddirection, the seat position control unit 105 rotates the left rearseat, on which the first occupant is seated, in the counterclockwisedirection, allowing the first occupant to continuously stare at theinterest object in a more comfortable position.

In addition, in consideration of the field of view of the secondoccupant blocked by the first occupant as the external interest objectis relatively moved, the display control unit 104 duplicates anddisplays the screen of an outer landscape viewed through the transparentdisplay of the left side window through the display of the right sidewindow, and the seat position control unit 105 rotates the right rearseat, on which the second occupant is seated, in the clockwise directionso that the second occupant can more easily and continuously stare atthe interest object through the display of the right side window.

When it is determined that the interest object is out of the ranges ofviews of the first occupant and the second occupant as the vehicletravels further forward, the seat position control unit 105 rotates theleft rear seat, on which the first occupant is seated, in the clockwisedirection, and rotates the right rear seat, on which the second occupantis seated, in the counterclockwise direction, to return the left andright rear seats to their original seat positions.

FIG. 5 is a view showing a method of controlling a vehicle based on anoccupant's intent according to another embodiment of the presentdisclosure.

The method of controlling a vehicle based on an occupant's intentaccording to the embodiment of the present disclosure includes: a stepS501 for obtaining gaze information of an occupant in a vehicle; a stepS502 for determining a point of interest located outside the vehicleusing the gaze information; and a step S503 for changing a seat positionof the occupant to enable a continuous stare at the point of interest asthe vehicle moves.

In the step S503, a seat on which the occupant is seated is rotated inconsideration of the driving direction and driving speed of the vehicle.

When the gaze information of a plurality of occupants is obtained in thestep S501, and the points of interest of the plurality of occupants aredetermined to be the same in the step S502, a seat of a first occupantis rotated, and a seat of a second occupant is slid forward or backwardand rotated, in the step S503.

The method of controlling a vehicle based on an occupant's intentaccording to the present disclosure further includes a step S504 forcontrolling an outer area, including the points of interest, which areviewed through a first side window close to the seating position of thefirst occupant, to be displayed through a second side window close tothe seating position of the second occupant, when the gaze informationof the plurality of occupants is obtained in the step S501 and thepoints of interest of the plurality of occupants are determined to bethe same in the step S502.

FIG. 6 is a view showing a vehicle display control system based on anoccupant's intent according to another embodiment of the presentdisclosure.

The vehicle display control system based on an occupant's intentaccording to the embodiment of the present disclosure includes: a firstoccupant gaze recognition unit 601 configured to recognize a gaze of afirst occupant in a vehicle; a second occupant gaze recognition unit 602configured to recognize a gaze of a second occupant in the vehicle; aunit for confirming an interest information display area crossover anddetermining priority 603 configured to determine interest informationfrom the gazes of the first occupant and the second occupant, to confirmwhether areas for displaying the interest information crossover witheach other on a windshield, and to determine a priority for display; anda display control unit 604 configured to control the display accordingto the priority for display.

The display control unit 604 performs display control in considerationof a predetermined initiative information when an interest informationdisplay area of the first occupant and an interest information displayarea of the second occupant overlaps with each other.

When the first occupant, not the second occupant, is set to have theinitiative, The display control unit 604 performs control to display theinterest information display area of the first occupant at a topposition while covering the interest information display area of thesecond occupant with the interest information display area of the firstoccupant.

When the first occupant, not the second occupant, is set to have theinitiative, the display control unit 604 performs control to move theinterest information display area of the second occupant and to allowthe interest information display area of the second occupant not tooverlap with the interest information display area of the firstoccupant.

The display control unit 604 performs control in consideration of theimportance of the interest information when the interest informationdisplay area of the first occupant and the interest information displayarea of the second occupant overlap with each other.

When the display control unit 604 determines that the interestinformation of the first occupant has the importance higher than that ofthe interest information of the second occupant, the display controlunit 604 performs control not to display the interest information of thesecond occupant, but to display the interest information of the firstoccupant.

In consideration of at least any one of the types of content displayedin a display area assigned to the first occupant and an operationsituation associated with the display area assigned to the firstoccupant, the display control unit 604 determines whether to display theinterest information of the second occupant in the display area assignedto the first occupant.

FIG. 7A and FIG. 7B are views showing vehicle windshield display controlfor each area, in consideration of priority, according to anotherembodiment of the present disclosure.

Referring to FIG. 7A, a driver A and a passenger B look at externalobjects through a front windshield of a vehicle.

At this time, when the driver A stares at a forward vehicle located inan area of W for a preset time or longer, driving-related information isdisplayed according to a result of tracking a gaze of the driver A, forexample, a distance from the forward vehicle, navigationinformation-based AR route guidance information, etc. are displayed in afirst area A-1 of the front windshield which is a transparent display.

The passenger B stares at a store located in an area of {circle around(2)} for the preset time or longer, and accordingly, the information ofthe store (e.g., summer sale, up to 70%) is displayed in a second areaB-1 of the front windshield, which is the transparent display.

Referring to FIG. 7B, the driver A looks at a parking tower (e.g., theparking tower is a destination set on a navigation route) located in thearea of {circle around (1)}, and accordingly, the information of theparking tower (e.g., parking cost per hour) is displayed in the secondarea B-1 of the front windshield, which is the transparent display.

At this time, as described above, a display area (that is, some of thesecond area B-1 of the front windshield, which is the transparentdisplay) for the information (e.g., summer sale, up to 70%) of the storelocated in the area of {circle around (2)} (e.g., summer sale, up to70%) according to the gaze of the passenger B and a display area for theinformation of the parking tower located in the area of {circle around(1)} according to the gaze of the driver A overlap with each other.

For example, the display control unit 604 performs control to displaythe external interest information area of the driver A and the externalinterest information area of the passenger B while overlapping the areaswith each other. At this time, on the contrary, as the gaze of thepassenger B stares at the store located in the first area A-1, whendisplaying the information about the store located in the first area A-1on the front windshield, which is the transparent display, if it isdetermined that the display area for the information will overlap withthe display area for the object at which driver A stares, the displaycontrol unit 604 performs control to display only the information aboutthe object, at which the driver A stares, but not to display theexternal interest information according to the gaze of the passenger B.

As another example, the display control unit 604 confirms that thedisplay priority of the information displayed on the transparent displayaccording to the gaze of driver A is higher than the display level ofthe information displayed on the transparent display according to thegaze of the passenger B, and displays the external interest informationarea of the driver A in front of the external interest information areaof the passenger B (i.e., some of the external interest information areaof the passenger B may be covered).

As still another example, the display control unit 604 confirms that thedisplay priority of the information displayed on the transparent displayaccording to the gaze of the driver A is higher than the display levelof the information displayed on the transparent display according to thegaze of the passenger B, and moves the display area for the externalinterest information of the passenger B so that the display area doesnot overlap with the display area for the external interest informationof the driver A.

For example, in the case of FIG. 7B, the display control unit 604performs control to move the display area for the information of thestore (e.g., summer sale, up to 70%) to the bottom right, so that thedisplay area does not overlap with the display area for the informationof the parking tower.

That is, when the driver A, not the passenger B, has the initiative, thedisplay control unit 604 performs control to display the display areafor the external interest information, at which the driver A stares, inpreference to the display area for the external interest information atwhich the passenger B stares.

In addition, when the information provided according to the gazerecognition of the occupant is driving, safety-related information(e.g., a lane, a speed limit, a traffic light, information about adistance from a forward vehicle, etc.), the display control unit 604performs control to display a display area for the external interestinformation in preference to other information (e.g., store information,etc.) display area.

FIG. 8A and FIG. 8B are views showing the vehicle windshield displaycontrol for each area, in consideration of priority during a relativemovement of an external interest object, according to the embodiment ofthe present disclosure.

Referring to FIG. 8A, a first occupant A and a second occupant B look anexternal object through a windshield of a left side window of a vehicle.

At this time, a music player is displayed in a first area A-1 accordingto the needs of the first occupant A, and an outline of the externalinterest object (Isolated Spire) is displayed in a second area B-1according to the needs of the second occupant B.

Referring to FIG. 8B, as the vehicle travels forward, when the interestinformation of the second occupant B invades a display area A-1 of thefirst occupant A, the display control unit 604 performs display controlfor the first area A-1 and the second area B-1 in consideration of theattributes of the content and the gaze monitoring results of the firstoccupant A.

Referring to FIG. 8B, when the content displayed in the display area A-1of the first occupant A is a content (e.g., music, weather, movies,etc.) that is not the external interest information based on the gazeinformation of the first occupant A, depending on the type of content orthe screen operation state of the first occupant A, the display controlunit 604 determines whether to display the information of the externalinterest object of the second occupant B on the first area A-1 while theinformation invades the first area A-1.

When the first occupant A selects music in the first area A-1, thedisplay control unit 604 performs control not to display the informationof the external interest object of the second occupant B in the firstarea A-1 so that the information does not interfere with the screenoperation of the first occupant A.

In addition, if the first occupant A is watching a movie in the firstarea A-1, the display control unit 604 performs control not to displaythe information of the external interest object of the second occupant Bin the first area A-1 so that the information does not interfere withthe first occupant A that watches a movie.

As another example, when the first occupant A is not performing anoperation on the content execution in the first area A-1, the displaycontrol unit 604 performs control to display the information of theexternal interest object of the second occupant B in the first area A-1.

In addition, if it is confirmed that the first occupant A is staring atanother place for a certain period of time or longer, as in {circlearound (3)}, the display control unit 604 performs control to displaythe information of the external interest object of the second occupant Bin the first area A-1.

FIG. 9 is a view showing a vehicle display control method based on anoccupant's gaze according to another embodiment of the presentdisclosure.

The vehicle display control method based on an occupant's gaze accordingto the embodiment of the present disclosure includes: a step S901 forrecognizing gaze information of a plurality of occupants in a vehicle; astep S902 for confirming interest information using the gazeinformation; a step S903 for confirming whether display areas of theinterest information displayed on a windshield being a transparentdisplay crossover with each other; and a step S904 for performingdisplay control when the display areas of the interest informationcrossover.

In the step S904, the display control is performed in consideration ofpredetermined initiative information according to the result that it isconfirmed that an interest information display area of the firstoccupant and an interest information display area of the second occupantoverlap with each other.

In the step S904, when the first occupant, not the second occupant, isset to have an initiative, the display control is performed such thatthe interest information display area of the first occupant is displayedat a top position while covering the interest information display areaof the second occupant.

In step S904, when the first occupant, not the second occupant, is setto have the initiative, the display control is performed such that theinterest information display area of the second occupant is moved sothat the interest information display area of the second occupant doesnot overlap with the interest information display area of the firstoccupant.

In the step S904, when the interest information display area of thefirst occupant and the interest information display area of the secondoccupant overlap with each other, the display control is performed inconsideration of the importance of the interest information.

In the step S904, when it is determined that the interest information ofthe first occupant the importance higher than that of the interestinformation of the second occupant, the display control is performedsuch that the interest information of the second occupant is notdisplayed, but the interest information of the first occupant isdisplayed.

In the step S904, in consideration of at least any one of the types ofcontent displayed in the display area assigned to the first occupant andthe operation situation associated with the display area assigned to thefirst occupant, it is determined whether to display the interestinformation of the second occupant in the display area assigned to thefirst occupant.

FIG. 10 is a view showing an external interest information display areadetermination and vehicle display control system based on an occupant'sgaze according to another embodiment of the present disclosure.

The external interest information display area determination and vehicledisplay control system based on an occupant's gaze according to theembodiment of the present disclosure includes: a first occupant gazerecognition unit 1001 configured to recognize a gaze of a first occupantin a vehicle; an external interest information confirmation unit 1003 ofthe first occupant configured to confirm an external interestinformation of the first occupant from the gaze of the first occupant; asecond occupant gaze recognition unit 1002 configured to recognize agaze of a second occupant in the vehicle; an external interestinformation confirmation unit 1004 of the second occupant configured toconfirm the external interest information of the second occupant fromthe gaze of the second occupant; an external interest informationdisplay area confirmation unit 1005 configured to confirm a display areafor external interest information of the first occupant and a displayarea for external interest information of the second occupant; and adisplay control unit 1006 configured to determine a priority fordisplay, in consideration of a recognition influence on the display areafor the external interest information of the first occupant by thedisplay area for the external interest information of the secondoccupant with relatively low information display importance.

When it is determined that the recognition influence is equal to orlower a predetermined value, the display control unit 1006 performsdisplay control to display the external interest information of thesecond occupant together with the external interest information of thefirst occupant.

When it is determined that the recognition influence exceeds thepredetermined value, the display control unit 1006 performs displaycontrol not to display the external interest information of the secondoccupant, but to display only the external interest information of thefirst occupant.

When it is determined that the recognition influence exceeds thepredetermined value, the display control unit 1006 performs displaycontrol to move the display area for the external interest informationof the second occupant so that the moved display area for the externalinterest information of the second occupant does not invade the displayarea for the external interest information of the first occupant.

The display control unit 1006 performs display control to display anoutline information about an original position of an external interestobject of the second occupant.

FIG. 11A to FIG. 11C are views showing external interest informationdisplay area determination and display control according to anotherembodiment of the present disclosure.

Referring to FIG. 11A to FIG. 11C, a driver A and a passenger B in avehicle look at external objects through a front windshield of thevehicle.

Referring to FIG. 11A, when the driver A stares at a forward vehiclelocated in the area of {circle around (1)} for a preset time or longer,driving-related information is displayed according to a result oftracking a gaze of the driver A, for example, a distance from theforward vehicle, navigation information-based AR route guidanceinformation, etc. are displayed in a first area A-1 of the frontwindshield configured as a transparent display. The passenger B staresat a store located in the area of {circle around (2)} for the presettime or longer, and accordingly, the information of the store (e.g.,summer sale, up to 70%) is displayed in a second area B-1 of the frontwindshield, which is a transparent display.

Referring to FIG. 11B, according to the gaze movement of the passengerB, a display area for the external interest information corresponding to{circle around (2)} is moved to the first area A-1.

At this time, the display control unit 1006 performs control to displaythe external interest information of the passenger B in the first areaA-1 when it is confirmed that the display area for the external interestinformation of the passenger B does not overlap with the display areafor the external interest information of the driver A.

Referring to FIG. 11C, according to the gaze movement of the driver A, arear side warning notification corresponding to {circle around (1)} isdisplayed as the external interest information of the driver A.

In this case, it is determined that the display area for the externalinterest information of the passenger B described in FIG. 11B overlapswith the display area for the external interest information of thedriver A, and the display control unit 1006 determines whether todisplay the external interest information of the passenger B.

At this time, the display control unit 1006 monitors the gazeinformation of the driver A and does not display the external interestinformation of the passenger B until the display area for the externalinterest information of the driver A does not overlap with the displayarea for the external interest information of the passenger B.

As another example, as shown in FIG. 11C, the display control unit 1006moves the display area for the external interest information ofpassenger B so that the display area for the external interestinformation of the passenger B does not overlap with the display areafor the external interest information of the driver A.

As shown in FIG. 11C, the display control unit 1006 moves the displayarea for the external interest information of the passenger B to thesecond area B-1 so that no obstacle interferes with the driver A toconfirm the external interest information.

The display control unit 1006 performs control to display the externalinterest information of the passenger B as AR or displayed as a cameravideo.

The display control unit 1006 continuously displays the position of theexternal interest object of the passenger B in the form of a dotted lineso that the position is recognized, and thus the display control unit1006 performs control for the position display not to interfere with thegaze of the driver A.

FIG. 12 is a view showing an external interest information display areadetermination and vehicle display control method based on an occupant'sgaze according to another embodiment of the present disclosure.

The external interest information display area determination and vehicledisplay control method based on an occupant's gaze according to theembodiment of the present disclosure includes: a step S1201 forrecognizing gaze information of a first occupant and a second occupantin a vehicle; a step S1202 for confirming a display area for externalinterest information of the first occupant and a display area forexternal interest information of the second occupant using the gazeinformation of the first occupant and the second occupant; and a stepS1203 for determining a priority for display and controlling the displayin consideration of an recognition influence on the display area for theexternal interest information of the first occupant by the display areafor the external interest information of the second occupant withrelatively low information display importance.

In the step S1203, when it is determined that the recognition influenceis equal to or lower than a predetermined value, the display control isperformed such that the external interest information of the secondoccupant is displayed together with the external interest information ofthe first occupant.

In step S1203, when it is determined that the recognition influenceexceeds the predetermined value, the display control is performed suchthat the external interest information of the second occupant is notdisplayed, but only the external interest information of the firstoccupant is displayed.

In step S1203, when it is determined that the recognition influenceexceeds the predetermined value, the display control is performed suchthat the display area for the external interest information of thesecond occupant is moved, and the moved display area for the externalinterest information of the second occupant does not invade the displayarea for the external interest information of the first occupant.

In step S1203, the display control is performed such that an outlineinformation about an original position of the external interest objectof the second occupant is displayed.

FIG. 13 is a view showing an occupant gaze departure determination andvehicle display control system according to another embodiment of thepresent disclosure.

The occupant gaze departure determination and vehicle display controlsystem according to the embodiment of the present disclosure includes:an occupant information recognition unit 1310 configured to recognizebehavioral information related to viewing a display screen of anoccupant in a vehicle; and a display control unit 1320 configured toperform control over the vehicle display using the behavior informationrelated to viewing the display screen, and the display control unit 1320analyzes the behavioral information related to viewing the displayscreen to determine that a gaze of the occupant departs from a displayscreen area, and accordingly, performs display control.

The occupant information recognition unit 1310 includes an occupant gazerecognition unit 1311 configured to obtain the gaze information of theoccupant in the vehicle and recognize whether the gaze of the occupantin the vehicle departs from the display screen area, and a departureduration time.

The display control unit 1320 performs control to blur the screen of avideo content when the duration time that the gaze of the occupantdeparts from the display screen area exceeds a preset time.

The occupant information recognition unit 1310 further includes a voicerecognition unit 1312 configured to obtain in-vehicle voice information,and the display control unit 1320 confirms that a call or in-vehicleconversation is in progress, using the in-vehicle voice information, andperforms control to pause for playback of video content corresponding tovideo playback.

The display control unit 1320 confirms that the call or in-vehicleconversation is in progress, using the in-vehicle voice information, andperforms control to lower a playback volume of the video content.

The display control unit 1320 performs control to display a wallpaperrather than the video content when the departure duration time lasts forthe preset time or longer.

The occupant information recognition unit 1310 further includes asitting status recognition unit 1313 configured to recognize theoccupant's sitting status for a seat in the vehicle, and the displaycontrol unit 1320 performs control to confirm the occupant approachingthe seat to auto-play the video content when a first occupant who hasthe initiative is seated on the seat.

When it is confirmed that a second occupant who has no initiative isseated on the seat, the display control unit 1320 assigns a portion ofthe display screen area to the second occupant.

The display control unit 1320 performs control to display the externalinterest information of the second occupant beyond the area assigned tothe second occupant, in consideration of at least any one ofauthorization setting information of the first occupant and boardingstatus information of the first occupant.

When the first occupant leaves the vehicle, the display control unit1320 performs control to display a function previously used by thesecond occupant on the screen in consideration of usage historyinformation of the second occupant.

FIGS. 14 to 17 are views showing an example of vehicle display controlin an occupant gaze departure situation according to another embodimentof the present disclosure.

Referring to FIG. 14 , when a gaze of an occupant (who has theinitiative) who is watching a video content being played on a vehicledisplay deviates from a screen, the display control unit 1320 performscontrol to blur the screen of the vehicle display so that the videocontent displayed on the screen does not interfere with the occupant.

At this time, when it is determined that the occupant makes a call, ortalks to another occupant in the vehicle, or is leaning on the seat withhis eyes closed, etc., that is, when it is determined that the occupanthas no intent to watch the video content, the display control unit 1320performs control to lower the playback volume of the video content.

At this time, when the video content corresponds to video playbackrather than real-time playback, such as TV content, the display controlunit 1320 performs control to pause the video content.

Referring to FIG. 15 , when the occupant that is watching the videocontent being played on the vehicle display moves to another seat ormoves, if the gaze movement of the occupant is within a reference value(e.g., if the gaze returns back into the screen area within 3 seconds)even if the gaze of the occupant is out of a preset range (e.g.: screenarea), the display control unit 1320 performs control to play back thevideo content as it is without changing the video content screen andplayback sound.

When the occupant leaves the seat, the display control unit 1320performs control to lower the playback volume of the video content, toblur the vehicle display screen (at this time, control to lower theplayback volume decreases sequentially over time from the moment thatthe occupant leaves the seat, and to gradually blur the screen), and topauses the video content if the video content corresponds to videoplayback rather than real-time playback.

If the occupant does not return to the seat even after the preset timefrom the moment that the occupant leaves the seat, the display controlunit 1320 performs control to convert the screen on which the videocontent is played into a wallpaper to display the wallpaper so that noother occupant cannot see the screen.

Referring to FIG. 16 , when detecting that the occupant who has left theseat is approaching the seat, while the wallpaper is being displayed onthe vehicle display, the display control unit 1320 performs control tostop the wallpaper display and to display the previously paused videocontent.

At this time, when it is confirmed that the occupant seated on the seathas the initiative, the display control unit 1320 performs control toauto-play the video content.

When the occupant who has approached the seat has the initiative, but isnot seated on the seat with only the occupant's gaze located on thescreen, the occupant clicks a play button to play the video content.

Referring to FIG. 17 , as the gaze of the occupant deviates from thevehicle display screen for a certain time or longer, the wallpaper isdisplayed.

At this time, when another occupant that has no initiative approachesthe seat, the display control unit 1320 assigns a portion of thecorresponding vehicle display screen area to another occupant.

At this time, when a screen lock is set, the display control unit 1320performs control not to display the video content that was previouslyplayed in the area assigned to the occupant having the initiative.

The area provided to another occupant (occupant B) who has no initiativeis shown as a second area B-1, and the area provided to the occupant(occupant A) who has the initiative is shown as a first area A-1.

When the external interest information of the occupant B is to bedisplayed in the first area A-1 beyond the second area B-1, the displaycontrol unit 1320 performs control to display even the external interestinformation of the occupant B in the first area A-1 according to theboarding status of the occupant A or whether the authority of theoccupant A having the initiative is set.

FIG. 18 is a view showing an example of vehicle display control upondeparture from control according to another embodiment of the presentdisclosure.

In a situation where the video content is played on the vehicle display,when the occupant having the initiative leaves the vehicle, the displaycontrol unit 1320 performs display control to display the wallpaper.

When the occupant having the initiative leaves the vehicle, the displaycontrol unit 1320 performs display control to display the functionspreviously used by another occupant as another occupant having noinitiative approaches the display.

FIG. 19 is a view showing an occupant gaze departure determination andvehicle display control method according to another embodiment of thepresent disclosure.

The occupant gaze departure determination and vehicle display controlmethod according to the embodiment of the present disclosure includes: astep S1901 for obtaining at least any one of gaze information, voiceinformation, and sitting status information of an occupant in a vehicle;and a step S1902 for confirming that the occupant's intent is to stopviewing a display screen, by using the information obtained in the stepS1901, and controlling a vehicle display.

In the step S1901, when the gaze information of the occupant in thevehicle is obtained, the gaze information including whether the gaze ofthe occupant in the vehicle departs from the display screen area and thedeparture duration time is obtained.

In the step S1902, when the departure duration time exceeds the presettime, control is performed such that the screen of the video content isblurred.

When the voice information is obtained in the step S1901, in the stepS1902, it is determined that a call or an in-vehicle conversation is inprogress, using the voice information, and control is performed suchthat the video content corresponding to the video playback is paused.

When the voice information is obtained in the step S1901, in the stepS1902, it is determined that the call or in-vehicle conversation is inprogress, using the voice information, and control is performed suchthat the playback volume of the video content is lowered.

In the step S1902, it is confirmed that the occupant's intent is to stopviewing the display screen, and control is performed such that thewallpaper, not the video content, is displayed.

When the sitting status information is recognized in the step S1901, thevideo content is automatically displayed depending on whether theoccupant seated in the seat in the step S1902 has been granted theinitiative.

When it is confirmed in the step S1901 that an occupant who has beengranted no initiative is seated, in the step S1902, a portion of thedisplay screen area is assigned to the occupant who has been granted noinitiative.

In the step S1902, in consideration of at least any one of theauthorization setting information of the occupant who has been grantedthe initiative and the boarding status information of the occupant whohas been granted the initiative, control is performed such that externalinterest information is displayed beyond the area assigned to theoccupant who has been granted no initiative.

In the step S1901, when it is confirmed that the occupant who has beengranted the initiative leaves the vehicle, and the occupant who has beengranted no initiative is seated, in the step S1902, control is performedsuch that the previously used functions are displayed on the screenusing the usage history information of the occupant who has been grantedno initiative.

FIG. 20 is a view showing a button display control in an occupantgaze-based vehicle display and the initiative control system accordingto another embodiment of the present disclosure.

The occupant gaze-based button display control and initiative controlsystem in a vehicle display according to the embodiment of the presentdisclosure includes: a first occupant gaze recognition unit 2001configured to recognize a gaze of a first occupant in a vehicle displayscreen area; and a button display control unit 2003 that performscontrol to display a button on an video content according to the gaze ofthe first occupant.

When the first occupant gaze recognition unit 2001 confirms that thegaze of the first occupant stays on a predetermined screen area for acertain period of time or longer, the button display control unit 2003generates and displays a cursor, and controls the movement and sizechange of the cursor according to the gaze movement of the firstoccupant and the gaze distance of the first occupant.

When the first occupant gaze recognition unit 2001 confirms that thegaze of the first occupant stays on the predetermined screen area for acertain period of time or longer, the button display control unit 2003performs control to display at least any one of a player button and aprogress button of the video content.

The occupant gaze-based button display control in the vehicle displayand initiative control system according to the embodiment of the presentdisclosure further includes a second occupant gaze recognition unit 2002configured to recognize a gaze of a second occupant participating in theuse of the vehicle display, and the button display control unit 2003performs control to display a button for the initiative switch.

The occupant gaze-based button display control and initiative controlsystem in the vehicle display according to the embodiment of the presentdisclosure further includes a display control unit 2004 configured toassign the vehicle display screen area to the first occupant and thesecond occupant when the initiative switch is approved.

The display control unit 2004, with the approval of the first occupant,also displays the external interest information of the second occupanteven in a first area assigned to the first occupant beyond a second areaassigned to the second occupant.

When the second occupant stares at the video content displayed in thefirst area for a preset time or longer, the display control unit 2004performs control to display the video content throughout the vehicledisplay screen area.

FIGS. 21 to 23 are views showing button display control according toanother embodiment of the present disclosure.

Referring to FIG. 21 , when it is confirmed that the occupant stares ata certain area on the screen of the video content played on the vehicledisplay, a necessary button is created.

The button display control unit controls the movement and size of thecursor based on the gaze movement of the occupant and the gaze distanceof the occupant.

The occupant selects a function item by staring at a certain area for apreset time or longer, or by inputting a gesture on a mobile phonescreen connected to a vehicle AVN.

The button display control unit displays the player button (play, pause,move forward, move backward, volume, screen brightness, etc.) and theprogress button, and executes a button display item for a functionexecution by item selection.

Referring to FIG. 22 , when the occupant stares at a left area on thescreen where the video content is played, a cursor is created, and thebutton display control unit changes the cursor to match the appearanceof the widget, confirms that the occupant stares at the left area forthe preset time or longer, and displays the player button and theprogress button.

The button display control unit provides a widget to float the button inan area where the gaze does not stay when the gaze of the occupantmoves.

Referring to FIG. 23 , the button display control unit determines thesize of the cursor differently, depending on the distance between theoccupant and the vehicle display.

FIG. 24 is a view showing initiative switch in another occupant'sparticipation according to another embodiment of the present disclosure.

In a situation where an occupant A is watching the video content playedon the vehicle display screen, when an occupant B participates in theuse of the vehicle display by staring at a particular area, a button forinitiative switch is displayed.

At this time, as the occupant B selects a button for initiative switch,the initiative is provided to the occupant B, or a button to watch thevideo together is displayed.

When the initiative switch is requested by the gaze of the occupant B, abutton for the occupant A, who currently has the initiative, to agree tothe initiative switch is provided on the screen.

When the initiative switch is approved at the request of the occupant B,only the screen of the occupant B may be configured, which will bedescribed with reference to FIG. 25 .

FIG. 25 is a view showing display control based on screen area divisionand gaze information upon another occupant's participation according toanother embodiment of the present disclosure.

The occupant A is watching the video content in a first area A-1 of thevehicle display, and according to the participation of the occupant B, asecond area B-1 of the vehicle display area is assigned to the occupantB.

It is assumed that in the first area A-1, the video content selected bythe occupant A is played, and in the second area B-1, the externalinterest information is displayed.

At this time, according to the approval of the occupant A, the externalinterest information is displayed on the first area A-1 beyond thesecond area B-1.

At this time, when the occupant B stares at the video content for acertain period of time or longer, it is confirmed that the occupant Bhas the intent to watch the video content instead of the externalinterest information, and control is performed such that the videocontent is displayed in the entire area of the vehicle display.

FIG. 26 is a view showing a method of button display control in anoccupant gaze-based vehicle display and initiative control according toanother embodiment of the present disclosure.

The method of button display control in an occupant gaze-based vehicledisplay and initiative control method according to the embodiment of thepresent disclosure includes: a step S2601 for recognizing a gaze of afirst occupant in a vehicle display screen area; and a step S2602 forperforming control to display a button on a video content according tothe gaze of the first occupant.

When it is confirmed in the step S2601 that the gaze of the firstoccupant stays on a predetermined screen area for a certain period oftime or longer, in the step S2602, a cursor is created and displayed,and the movement and size change of the cursor are controlled accordingto the gaze movement of the first occupant and the gaze distance of thefirst occupant.

When it is confirmed in the step S2601 that the gaze of the firstoccupant stays on the predetermined screen area for a certain period oftime or longer, in the step S2602, control is performed such that atleast any one of a player button and a progress button of the videocontent is displayed.

In the step S2601, a gaze of a second occupant participating in the useof the vehicle display is additionally recognized, and in the stepS2602, control is performed such that a button for initiative switch isdisplayed.

The occupant gaze-based button display control in the occupantgaze-based vehicle display and initiative control method according tothe embodiment of the present disclosure further includes a step S2603of confirming the approval of the first occupant according to aninitiative switching request of the second occupant, and performinginitiative switching control.

The occupant gaze-based button display control based on the occupantgaze-based vehicle display and the initiative control method accordingto the embodiment of the present disclosure further includes a stepS2604 of splitting and assigning the vehicle display screen area to thefirst occupant and the second occupant when the initiative switchingrequest is approved, and performing display control using the gazeinformation of the first occupant and the second occupant.

In the step S2604, with the approval of the first occupant, the externalinterest information of the second occupant is also displayed in thefirst area assigned to the first occupant beyond the second areaassigned to the second occupant.

In the step S2604, when the second occupant stares at the video contentdisplayed in the first area for a preset time or longer, control isperformed such that the video content is displayed throughout thevehicle display screen area.

FIG. 27 is a view showing a vehicle display control system fortransmitting external interest information according to embodiment ofthe present disclosure.

The vehicle display control system for transmitting external interestinformation according to the embodiment of the present disclosureincludes: an external interest information confirmation unit 2701configured to confirm external interest information of a first occupantin a vehicle; and a display control unit 2703 configured to performcontrol to display the external interest information of the firstoccupant through a vehicle display.

The external interest information confirmation unit 2701 uses gazeinformation of the first occupant to confirm an external interestobject.

The external interest information confirmation unit 2701 uses voiceinformation and the gaze information of the first occupant to confirmthe external interest object.

The external interest information confirmation unit 2701 confirms theexternal interest object using a gesture information and the gazeinformation of the first occupant.

The display control unit 2703 performs control to display the externalinterest information through at least any one of highlight, areadisplay, contour display, and zoom display.

The vehicle display control system for transmitting external interestinformation according to the embodiment of the present disclosurefurther includes a boarding position recognition unit 2702 configured toconfirm position information of the first occupant and a second occupantin the vehicle, and the display control unit 2703 performs control todisplay the external interest information through a display arearecognizable by the second occupant in consideration of the positioninformation.

FIG. 28A and FIG. 28B are views showing external interest informationdisplay and external interest information display area control accordingto another embodiment of the present disclosure.

Referring to FIG. 28A, a driver A recognizes an interest object (e.g.,bear) viewed through the front windshield of the driving vehicle.

The external interest information confirmation unit confirms theexternal interest information when the gaze of the driver A stays on acertain area for a certain period of time or longer.

When the driver A says “Did you see that, guys?”, the external interestinformation confirmation unit checks the external interest informationbased on the utterance information and gaze information of the driver A.

The external interest information confirmation unit confirms theexternal interest information when the gaze of the driver A stays on acertain area for a certain period of time or longer and performs apredetermined gesture (e.g., finger pointing operation) for indicatingthe external interest information.

The display control unit performs control to display the externalinterest information of the driver A through at least any one ofhighlight, area display, contour display, magnification display.

Referring to FIG. 28A, the display control unit performs control todisplay the area of the external interest object of the driver A throughthe area display on the front windshield, which is a transparentdisplay.

Referring to FIG. 28B, as described above, when the external interestinformation of the driver A is displayed on the front windshield, it isassumed that it is difficult for a passenger B and passenger C, whichare blocked by the driver A and located at the rear seat of the driverA, to easily confirm the corresponding external interest information.

At this time, in consideration of the seating position information ofthe passengers obtained through the boarding position recognition unit,the display control unit performs control to display the externalinterest information on the transparent display of the left side windowlocated near the sitting position of the passenger B and to display theexternal interest information on the transparent display of the rightside window located near the sitting position of the passenger C, sothat the passenger B and the passenger C can easily confirm the externalinterest information.

The boarding position recognition unit recognizes the position of theoccupant in the vehicle using at least any one of a seat sensor and anindoor camera, and the display control unit uses the position todetermine a display area (the aforementioned rear seat left side windowdisplay area and rear seat right side window display area) that thepassengers can comfortably see, and displays the external interestinformation in the determined area.

FIG. 29 is a view showing a vehicle display control method fortransmitting external interest information according to anotherembodiment of the present disclosure.

The vehicle display control method for transmitting external interestinformation according to the embodiment of the present disclosureincludes: a step S2901 for confirming external interest information of afirst occupant in a vehicle; a step S2902 for confirming the boardingpositions of the first occupant and the second occupant in the vehicle;and a step S2903 for performing control to display the external interestinformation through a vehicle display in consideration of the boardingpositions.

In the step S2901, the external interest information is confirmed usingat least any one of gaze information, voice information, and gestureinformation of the first occupant.

In the step S2903, control is performed such that the external interestinformation is displayed through at least any one of highlight, areadisplay, contour display, or zoom display.

In the step S2903, when it is confirmed by using the boarding positionthat the field of view of the second occupant is not blocked, theexternal interest information confirmed by the first occupant beyond afirst display area is displayed in the first display area.

In the step S2903, when it is confirmed by using the boarding positionthat the field of view of the second occupant is blocked, the externalinterest information confirmed by the first occupant beyond the firstdisplay area is displayed in a second display area that is differentfrom the first display area.

FIG. 30 is a view showing a vehicle display control system forsynthesizing and displaying external interest information according toanother embodiment of the present disclosure.

The vehicle display control system for synthesizing and displayingexternal interest information according to the embodiment of the presentdisclosure includes: an external interest information confirmation unit3001 configured to confirm an interest object outside a vehicle in whichan occupant in the vehicle is interested; an occupant operationinformation recognition unit 3002 configured to recognize operationinformation of the occupant in the vehicle associated with the rotationof external interest information; and a display control unit 3003configured to synthesize and display a 3D image for the interest objectoutside the vehicle, and to rotate and display the 3D image according tothe operation information.

The external interest information confirmation unit 3001 confirms aninterest object located in an area where the gaze of the occupant in thevehicle stays for a preset time or longer.

The occupant operation information recognition unit 3002 recognizes atleast any one of a touch input, gesture input, and voice input as theoperation information.

The display control unit 3003 determines the degree of rotation of the3D image according to the operation information, rotates the 3D image todisplay the rotated 3D image, and displays the 3D image by restoring the3D image to the actual position of the interest object when there is nofurther operation for a preset time or longer.

FIGS. 31A, 31B and 32 are views showing external interest informationrotation and display according to another embodiment of the presentdisclosure.

Referring to FIG. 31A and FIG. 31B, the gaze information of the occupantin the vehicle is used to determine the external interest object amongthe objects viewed through the vehicle windshield, and the correspondinginterest object is synthesized into a 3D form and displayed.

At this time, the external interest information synthesized in 3D formis displayed in the front windshield area, which is a transparentdisplay.

When the occupant in the vehicle drags the external interest informationsynthesized in 3D form to the right using touch or gesture, the externalinterest information synthesized in 3D form is rotated to the right bythe value of the drag.

When a preset time passes after the external interest informationsynthesized in 3D form is rotated and displayed according to theoperation of the occupant in the vehicle, the position of the externalinterest information synthesized in the 3D form is restored to theposition of the actual external interest object.

Referring to FIG. 32 , the external interest object is synthesized anddisplayed in 3D form, and control is performed such that the 3D objectis rotated by 360 degrees up and down/left and right through multimodalinteractions such as touch, gesture, and voice recognition of theoccupant in the vehicle.

FIG. 33 is a view showing a vehicle display control method forsynthesizing and displaying external interest information according toanother embodiment of the present disclosure.

The vehicle display control method for synthesizing and displayingexternal interest information according to the embodiment of the presentdisclosure includes: a step S3301 for confirming external interestinformation of an occupant in a vehicle; a step S3302 for recognizingoperation information of the occupant associated with the rotation ofthe external interest information; and a step S3303 for performingcontrol to synthesize and display the external interest information as3D image, and to rotate and display the 3D image according to theoperation information.

In the step S3301, the external interest information for an externalobject located in an area where the gaze of the occupant in the vehiclestays for a preset time or longer is confirmed.

In the step S3302, at least any one operation information of touchinput, gesture input, and voice input is recognized.

In the step S3303, the 3D image is rotated and displayed according tothe degree of rotation determined based on the operation information,and when there is no further operation for a preset time or longer, the3D image is displayed by restoring the 3D image to the actual positionof the interest object.

FIG. 34 is a view showing a vehicle display control system for storingand sharing external interest information according to anotherembodiment of the present disclosure.

The vehicle display control system for storing and sharing externalinterest information according to the embodiment of the presentdisclosure includes: an occupant operation recognition unit 3401configured to receive operation information for an video storage requestfor an external interest object of a vehicle occupant; and a displaycontrol unit 3402 configured to perform video storage according to theoperation information for the external interest object at which thevehicle occupant stares through a vehicle windshield configured as atransparent display.

The occupant operation recognition unit 3401 receives the operationinformation that is input using at least any one of gaze information,voice information, gesture information, and touch information.

The display control unit 3402 performs control to display informationabout the external interest object as AR information.

The display control unit 3402 captures and stores an external foregroundand the AR information at the same time.

The display control unit 3402 controls the display for screen capture bywhite-out during a preset time.

The vehicle display control system for storing and sharing externalinterest information according to the embodiment of the presentdisclosure further includes: an external interest informationtransmission unit 3403 configured to transmit a video stored accordingto the operation information to at least any one of a divided area in avehicle display, a mobile terminal of the vehicle occupant, and a mobileterminal of another user.

The display control unit 3402 performs control to divide and assign adisplay area to a plurality of occupants in the vehicle, store theexternal interest information video according to the video storagerequest made within each area, and display the stored video in apredetermined area of an assigned display area.

The display control unit 3402 transmits control information to move thetransparent display in consideration of access control priorities of theplurality of occupants.

The display control unit 3402 performs control to move the videoinformation stored in a predetermined area of a first occupant to apredetermined area of a second occupant according to the occupantrequest and sharinga of setting information.

Hereinafter, in FIGS. 35 to 38 , it is assumed that a rear seat leftside occupant in a vehicle stares at the outside through a rear seatleft side window transparent display.

FIG. 35A and FIG. 35C are views showing a process of capturing theexternal interest information according to another embodiment of thepresent disclosure.

Referring to FIG. 35A, the display control unit 3402 displays fixedinformation including at least any one of position information, weather,battery information, and transparency of a transparent display throughthe transparent display.

Referring to FIG. 35B, the occupant operation recognition unit 3401confirms the area where the gaze of the occupant stays for a certainperiod of time or longer through the rear seat left side window, and thedisplay control unit 3402 displays AR information based on the gazetracking information.

At this time, the display control unit 3402 displays the information ofa mountain name and height, weather, distance, and travel time (the timerequired for the vehicle to travel from the current position to theentrance to the mountain, the time required to climb the mountain fromthe entrance to the mountain to the top of the mountain, etc.) as ARinformation.

Referring to FIG. 35C, the occupant operation recognition unit 3401recognizes various pieces of operation information of the occupant todetermine the intent of storing the external interest information.

That is, the occupants request the storage of the corresponding externalinterest information in a variety of ways, including gaze recognitionand a touch of a save button within a mobile device app, gazerecognition and a bottom swipe touch within the mobile device app, gazerecognition and a specific voice command (e.g., save it or capturethis), gaze recognition and a specific gesture, and gaze recognition anda display swipe down touch.

Referring to FIG. 35B, the display control unit 3402 simultaneouslycaptures the external foreground and AR information (as another example,it is also possible to cut and store only the area of the correspondingexternal interest information) according to the storage request receivedfrom the occupant operation recognition unit, and in this case, thecapture information may include a photograph, video, AR information, GPSposition information.

The display control unit 3402 whites the screen out for a preset time(e.g., 1 second) when capturing the screen.

The display control unit 3402 performs control to display the screenshotscreen in a specific area of the display (e.g., bottom left) with acertain size (e.g., 15 cm×15 cm) for a certain period of time (e.g., 3seconds).

FIG. 36A and FIG. 36B are views showing a process of sharing theexternal interest information with a smartphone according to anotherembodiment of the present disclosure.

According to the gaze tracking of the occupant, in a situation where theexternal interest information is displayed in AR through the rear seatside window transparent display, when the occupant operation recognitionunit 3401 receives information that the occupant touches a particularbutton using his/her mobile terminal connected to the vehicle, thedisplay control unit 3402 captures a screen according to a preset method(e.g., captures the external foreground and AR information at the sametime), and when captured, the screen is whited out for the preset time(e.g., 1 second).

The external interest information transmission unit 3403 transmits thecaptured screen to the mobile terminal of the occupant, and the occupantcan check whether a desired capture image has been received normally,through the app of the mobile terminal (e.g., album).

Through in-album manipulation, the occupant can flip through the imagedisplayed in the mobile terminal, and presses a specific button (sharebutton) to transmit the image to be shared to someone else's smartphoneor a vehicle.

FIG. 37A to FIG. 37C are views showing a process of storing andconfirming external interest information for each occupant throughtransparent display sliding according to another embodiment of thepresent disclosure.

Referring to FIG. 37A, an occupant A is seated on the rear left seat ofthe driving vehicle, an occupant B is seated on the rear right seat ofthe driving vehicle, and the occupant A and the occupant B stare at theoutside through the rear seat side window.

The rear seat side window is configured as a transparent display andprovides an area assigned for each occupant.

For example, in consideration of the viewing angle of the occupant, afirst area A-1 is assigned to the occupant A and a second area B-1 isassigned to the occupant B.

As another example, it is possible to assign a display area based on theposition of the external object at which the occupants currently stare.In other words, as an example different from that shown in FIG. 37A,when the occupant A stares at the isolated spire as the interest object,and the occupant B stares at the double spire as the interest object, itis also possible to assign the second area to the occupant A and thefirst area to the occupant B.

Referring to the illustrations in FIG. 37A, the occupant A stares at thedouble spire shown beyond the first area A-1 as the interest object, andwhen the occupant A performs swipe down on the display, the displaycontrol unit 3402 stores and displays the captured image in a specificpersonalization area at the bottom of the display area. In addition,since the occupant B is a person who is further away than the occupant Afrom the rear seat left side window, the occupant B transmits thecapture intent through the gaze recognition and air gesture, and thedisplay control unit 3402 stores and displays the capture image asrequested by the occupant B in a specific personalization area at thebottom of the display area.

Referring FIG. 37B, it is assumed that in an autonomous drivingsituation, the driving control depends on the autonomous driving mode,an occupant C seated on the driver's seat is viewing the outside scenerythrough the left side window, and the occupant C has the left-sidewindow display access control with a priority higher than that that ofthe occupant B (It is assumed that the priority for display accesscontrol is higher, the closer the display from the boarding position).

At this time, the occupant C slides forward the transparent displaycorresponding to the second area B-1, and thus, the second area B-1 isdisposed at the left side of the occupant C, and the second area B-1 isassigned to occupant C as a third area C-1.

The occupant C requests the storage of the external interest informationthrough a display swipe down, and the display control unit 3402 displaysthat the external interest information is stored as a capture image in aspecific area at the bottom of the display.

Referring to FIG. 37C, in consideration of a face ID camera placed inthe vehicle and the seat position information, authentication isperformed for each occupant, and when authentication is completed, apersonalized content history is displayed in an area at the bottom ofthe display area assigned to each occupant.

The content history display displays the image/video files in the orderin which they are captured.

FIG. 38A to FIG. 38C are views showing the sharing of the externalinterest information between displays according to another embodiment ofthe present disclosure.

Referring to FIG. 38A to FIG. 38C, a first area A-1 is assigned to anoccupant A and a second area B-1 is assigned to an occupant B, and thecontent history of each occupant is displayed in a specific area at thebottom of each area.

Referring to FIG. 38A, the occupant B selects a certain content in thecontent history (long press, which can be selected through gesture), anddrags the object and requests the transfer of the object to the displayof the occupant A.

Referring to FIG. 38B, in consideration of the information sharingauthorization setting information of the occupant A, when theinformation sharing is approved, the display control unit 3402 moves thecorresponding particular content to the personalized area of theoccupant A and displays it.

At this time, the display control unit 3402 performs control to allownot only the same side (rear seat left side window) display but thefront or opposite side (rear seat rear side window) display based on theoccupant's request and information sharing setting to share the externalinterest information according to the directionality of the drag anddrop.

Referring to FIG. 38C, a particular content shared by the operation ofthe occupant B is added and stored into the personalization area of theoccupant A.

FIG. 39 is a view showing a vehicle display control method for storingand sharing external interest information according to anotherembodiment of the present disclosure.

The vehicle display control method for storing and sharing externalinterest information according to the embodiment of the presentdisclosure includes: a step S3901 for receiving a video storage requestfor an external interest object of a vehicle occupant; and a step S3902for performing a video storage for the external interest object at whichthe vehicle occupant stares through a vehicle windshield configured as atransparent display and performing display control.

In the step S3901, a video storage request input using at least any oneof gaze information, voice information, gesture information, and touchinformation is received.

In the step S3902, display control is performed such that informationabout the external interest object is displayed as AR information.

In the step S3902, an external foreground and AR information arecaptured and stored at the same time.

In the step S3902, display control for screen capture is performed bywhite-out a screen during a preset time.

The vehicle display control method for storing and sharing externalinterest information according to the embodiment of the presentdisclosure further includes a step S3903 for sharing the externalinterest information by transmitting the stored video to at least anyone of a divided area in a vehicle display, a mobile terminal of thevehicle occupant, and a mobile terminal of another user.

In the step S3902, control is performed such that the display area isdivided and assigned to a plurality of occupants in the vehicle, and anexternal interest information video is stored according to the videostorage request made within each area, and the stored video is displayedin a predetermined area of the assigned display area.

In the step S3902, in consideration of access control priorities of aplurality of occupants. display control is performed such that thetransparent display is moved.

In the step S3902, control is performed such that video informationstored in a predetermined area of a first occupant is moved to apredetermined area of a second occupant according to the occupantrequest and sharing

setting information.

FIG. 40 is a view showing a vehicle display area control system forexternal interest information according to another embodiment of thepresent disclosure.

The occupant gaze-based external interest information display areadetermination and vehicle display control system according to theembodiment of the present disclosure includes: an external interestinformation display area confirmation unit 4001 configured to confirm adisplay area for displaying external interest information of a pluralityof occupants in the vehicle; and a display control unit 4002 configuredto adjust the display area of the external interest information, inconsideration of at least any one of importance of the external interestinformation and the initiative granted to the plurality of occupants.

The display control unit 4002 moves the display area of the externalinterest information of a second occupant so that the display area ofthe external interest information of the second occupant does notoverlap with the display area of the external interest information of afirst occupant, when it is determined that the importance of theexternal interest information of the first occupant is higher than theimportance of the external interest information of the second occupant.

The display control unit 4002 moves the display area of the externalinterest information of the second occupant when the first occupant hasthe initiative and it is determined that the display area of theexternal interest information of the first occupant and the display areaof the external interest information of the second occupant overlap witheach other in a certain part or more.

The display control unit 4002 performs control to display outlineinformation about the original position of the external interestinformation of the second occupant.

In consideration of the gaze information of the first occupant, thedisplay control unit 4002 restores the display area of the externalinterest information of the second occupant to the area before moving.

FIG. 41A to FIG. 41C are views showing a process of controlling avehicle display area for external interest information according toanother embodiment of the present disclosure.

Referring to FIG. 41A, a driver A and a passenger B look at externalobjects through a front windshield of a vehicle.

At this time, when the driver A stares at a forward vehicle located inthe area of {circle around (1)} for a preset time or longer,driving-related information (e.g., distance from the forward vehicle andAR route guidance information based on navigation information) isdisplayed on the front windshield, which is a transparent display,according to the result of tracking the gaze of the driver A.

The passenger B stares at a store (café) located in the area of {circlearound (2)} for a preset time or longer, and accordingly, theinformation about the store (e.g., phone number, the number of floors,menu) is displayed on the front windshield, which is a transparentdisplay.

Referring to FIG. 41B, as the driver A looks at a stopped vehiclelocated in the area of {circle around (1)}, the display area for theexternal interest information of the passenger B overlaps with thedisplay area for the external interest information of the driver A.

The display control unit 4002 changes the position of the display areafor the external interest information of the passenger B in order toprevent the field of view of the initiative owner, the driver A, frombeing blocked by this overlapping phenomenon.

That is, while the passenger B is staring at the information about thecafé located on the first floor of the building, the driver A isprovided with rear-facing warning information, and when the driver Astares at another vehicle, according to the priority of the initiativeand information, the display control unit 4002 performs control to moveand display the café information to the upper area of the building.

At this time, the display control unit 4002 continuously displays theposition of the previous interest information in the form of a dottedline, so that the position can be recognized.

Referring to FIG. 41C, when the rear side warning information providedto the driver A is no longer transmitted, or when the driver A moveshis/her gaze to another object (forward vehicle), the display controlunit 4002 restores the position of the display area for the externalinterest information of the passenger B.

FIG. 42 is a view showing a method of controlling a vehicle display areafor external interest information according to another embodiment of thepresent disclosure.

The occupant gaze-based external interest information display areadetermination and a vehicle display control method according to theembodiment of the present disclosure includes: a step S4201 forconfirming an area for displaying external interest information of anoccupant in a vehicle; and a step S4202 for adjusting the position ofthe area for displaying the external interest information, inconsideration of at least any one of importance of the external interestinformation and information about the initiative granted to theoccupant.

In the step S4202, when it is determined that the importance of theexternal interest information of a first occupant is higher than theimportance of the external interest information of a second occupant,the positions of the display area of the external interest informationof the second occupant is moved, so that the display area of theexternal interest information of the second occupant does not overlapwith the display area of the external interest information of the firstoccupant.

In the step S4202, when the first occupant has the initiative, and it isconfirmed that the display area for the external interest information ofthe first occupant and the display area for the external interestinformation of the second occupant overlap with each other in a certainpart or more, the display area for the external interest information ofthe second occupant is moved.

In the step S4202, control is performed such that outline informationabout the original position of the external interest information of thesecond occupant is displayed.

In the step S4202, in consideration of the gaze information of the firstoccupant, the display area for the external interest information of thesecond occupant is restored to the area before moving.

FIG. 43 is a view showing a vehicle display control system fordisplaying and storing external interest information according toanother embodiment of the present disclosure.

The vehicle display control system for displaying and storing externalinterest information according to the embodiment of the presentdisclosure includes: an external interest object confirmation unit 4301configured to confirm an external interest object in which an occupantin a driving vehicle is interested; and a display control unit 4302configured to perform control to display external interest informationabout the external interest object in a first display area, to move anddisplay the external interest information to a second display areaaccording to a driving situation, and to store the external interestinformation in an external interest information storing unit 4303.

The external interest object confirmation unit 4301 confirms theexternal interest object using at least any one of gaze information,voice information, and gesture information of the occupant.

When the gaze of the occupant departs from a display area for theexternal interest information, the display control unit 4302 performscontrol to simplify the external interest information, move thesimplified external interest information to the second display area,which is a side window display, and to display the simplified externalinterest information.

The display control unit 4302 performs control such that the simplifiedexternal interest information is accumulated in chronological order anddisplayed in the second display area.

The vehicle display control system for displaying and storing externalinterest information according to the embodiment of the presentdisclosure further includes a destination setting request receiving unit4304 configured to receive a destination setting request for thesimplified external interest information.

The vehicle display control system for displaying and storing externalinterest information according to the embodiment of the presentdisclosure further includes a route setting unit 4305 configured toestablish a route to an external point of interest selected inaccordance with the destination setting request.

FIG. 44A and FIG. 44B are views showing external interest informationdisplay according to another embodiment of the present disclosure.

Referring to FIG. 44A, a driver in a driving vehicle stares at aninterest object (parking lot) for a preset time or longer among externalobjects viewed through a front windshield configured as a transparentdisplay, and accordingly, external interest information (e.g.,information about the name, position and cost of the parking lot) isdisplayed as AR information.

At this time, the display control unit 4302 performs control to confirmthe external interest object using voice information, head movementinformation, and gesture information in addition to occupant gazeinformation, and to display information about the external interestobject.

Referring to FIG. 44A, the display control unit 4302 simplifies thecorresponding external interest information (e.g., simplifiedinformation such as the distance from the current position, parkingcosts, etc.) when the driver's field of view departs from a display areafor the external interest information, or when the external interestobject becomes out of the field of view of the driver as the drivingvehicle moves, and displays the simplified external interest informationin the form of a log.

At this time, the display control unit 4302 performs control to displaythe simplified external interest information in a predetermined area ofa front display or a side window display.

Referring to FIG. 44B, when the external point of interest movesrelatively and is out of the driver's field of view as the vehicletravels, the display control unit 4302 displays the external interestinformation on the side window display by time. That is, the displaycontrol unit 4302 accumulates and displays the external interestinformation in the order of time elapsed.

At this time, the display control unit 4302 performs control to displayinformation about the external interest object, located on the frontleft side of the driving vehicle, through the left side window display,and to display the information about the external interest object,located on the front right side of the driving vehicle, through theright side window display.

When receiving a destination setting request for particular externalinterest information among the plurality of stored external interestinformation, the route setting unit 4305 sets a path from the currentposition to the position of the corresponding external interestinformation.

The destination setting request receiving unit 4304 receives adestination setting request using touch, voice, air gesture, and amobile device for external interest information stored in the sidewindow display.

FIG. 45 is a view showing a vehicle display control method fordisplaying and storing external interest information according toanother embodiment of the present disclosure.

The vehicle display control method for displaying and storing externalinterest information according to the embodiment of the presentdisclosure includes: a step S4501 for confirming an external interestobject of a driving vehicle occupant; and a step S4502 for performcontrol to display external interest information for the externalinterest object in a first display area, and moving the externalinterest information to a second display area according to a drivingsituation to display the moved external interest information.

In the step S4501, the external interest object is confirmed using atleast any one of the gaze information, voice information, and gestureinformation of the occupant.

In the step S4502, when the gaze of the driving vehicle occupant departsfrom the display area for the external interest information, control isperformed such that the external interest information is simplified, andthe simplified external interest information is moved to the seconddisplay area which is the side window display and displayed.

In the step S4502, control is performed such that the simplifiedexternal interest information is accumulated in chronological order andis displayed in the second display area.

The vehicle display control method for displaying and storing externalinterest information according to the embodiment of the presentdisclosure further includes a step S4503 for receiving a destinationsetting request for the simplified external interest information andsetting a route to an external point of interest selected according tothe destination setting request.

On the other hand, the method according to an embodiment of the presentdisclosure may be implemented in a computer system or recorded in arecording medium. The computer system may include at least one or moreprocessor, memory, user input device, data communication bus, useroutput device, and storage. Each of the aforementioned componentsperforms data communication via a data communication bus.

The computer system may further include a network interface coupled tothe network. The processor may be a central processing unit (CPU) or maybe a semiconductor device that processes instructions stored in thememory and/or storage.

The memory and storage may include various types of volatile ornon-volatile storage media. Examples of the memory may include ROM andRAM.

Thus, the method according to an embodiment of the present disclosuremay be implemented in a manner that is executable on a computer. Whenthe method according to an embodiment of the present disclosure isperformed on a computer device, computer-readable instructions mayperform the method according to the present disclosure.

On the other hand, the method according to the present disclosuredescribed above can be implemented as a computer-readable code on acomputer-readable recording medium. The computer-readable recordingmedia includes all kinds of recording media that store data that can bedecrypted by a computer system. Examples thereof may include read onlymemory (ROM), random access memory (RAM), magnetic tape, magnetic disk,flash memory, optical data storage, and the like. In addition, thecomputer-readable recording medium may be distributed to a computersystem connected by a computer communication network, and stored andexecuted as code that can be read in a distributed manner.

The components described in the example embodiments may be implementedby hardware components including, for example, at least one digitalsignal processor (DSP), a processor, a controller, anapplication-specific integrated circuit (ASIC), a programmable logicelement, such as an FPGA, other electronic devices, or combinationsthereof. At least some of the functions or the processes described inthe example embodiments may be implemented by software, and the softwaremay be recorded on a recording medium. The components, the functions,and the processes described in the example embodiments may beimplemented by a combination of hardware and software.

The method according to example embodiments may be embodied as a programthat is executable by a computer, and may be implemented as variousrecording media such as a magnetic storage medium, an optical readingmedium, and a digital storage medium.

Various techniques described herein may be implemented as digitalelectronic circuitry, or as computer hardware, firmware, software, orcombinations thereof. The techniques may be implemented as a computerprogram product, i.e., a computer program tangibly embodied in aninformation carrier, e.g., in a machine-readable storage device (forexample, a computer-readable medium) or in a propagated signal forprocessing by, or to control an operation of a data processingapparatus, e.g., a programmable processor, a computer, or multiplecomputers. A computer program(s) may be written in any form of aprogramming language, including compiled or interpreted languages andmay be deployed in any form including a stand-alone program or a module,a component, a subroutine, or other units suitable for use in acomputing environment. A computer program may be deployed to be executedon one computer or on multiple computers at one site or distributedacross multiple sites and interconnected by a communication network.

Processors suitable for execution of a computer program include, by wayof example, both general and special purpose microprocessors, and anyone or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read-only memory ora random access memory or both. Elements of a computer may include atleast one processor to execute instructions and one or more memorydevices to store instructions and data. Generally, a computer will alsoinclude or be coupled to receive data from, transfer data to, or performboth on one or more mass storage devices to store data, e.g., magnetic,magneto-optical disks, or optical disks. Examples of informationcarriers suitable for embodying computer program instructions and datainclude semiconductor memory devices, for example, magnetic media suchas a hard disk, a floppy disk, and a magnetic tape, optical media suchas a compact disk read only memory (CD-ROM), a digital video disk (DVD),etc. and magneto-optical media such as a floptical disk, and a read onlymemory (ROM), a random access memory (RAM), a flash memory, an erasableprogrammable ROM (EPROM), and an electrically erasable programmable ROM(EEPROM) and any other known computer readable medium. A processor and amemory may be supplemented by, or integrated into, a special purposelogic circuit.

The processor may run an operating system (OS) and one or more softwareapplications that run on the OS. The processor device also may access,store, manipulate, process, and create data in response to execution ofthe software. For purpose of simplicity, the description of a processordevice is used as singular; however, one skilled in the art will beappreciated that a processor device may include multiple processingelements and/or multiple types of processing elements. For example, aprocessor device may include multiple processors or a processor and acontroller. In addition, different processing configurations arepossible, such as parallel processors.

Also, non-transitory computer-readable media may be any available mediathat may be accessed by a computer, and may include both computerstorage media and transmission media.

The present specification includes details of a number of specificimplements, but it should be understood that the details do not limitany invention or what is claimable in the specification but ratherdescribe features of the specific example embodiment. Features describedin the specification in the context of individual example embodimentsmay be implemented as a combination in a single example embodiment. Incontrast, various features described in the specification in the contextof a single example embodiment may be implemented in multiple exampleembodiments individually or in an appropriate sub-combination.Furthermore, the features may operate in a specific combination and maybe initially described as claimed in the combination, but one or morefeatures may be excluded from the claimed combination in some cases, andthe claimed combination may be changed into a sub-combination or amodification of a sub-combination.

Similarly, even though operations are described in a specific order onthe drawings, it should not be understood as the operations needing tobe performed in the specific order or in sequence to obtain desiredresults or as all the operations needing to be performed. In a specificcase, multitasking and parallel processing may be advantageous. Inaddition, it should not be understood as requiring a separation ofvarious apparatus components in the above described example embodimentsin all example embodiments, and it should be understood that theabove-described program components and apparatuses may be incorporatedinto a single software product or may be packaged in multiple softwareproducts.

What is claimed is:
 1. A system for controlling a vehicle display basedon occupant's gaze, comprising: a first occupant gaze recognition unitconfigured to recognize a gaze of a first occupant in a vehicle; asecond occupant gaze recognition unit configured to recognize a gaze ofa second occupant in the vehicle; an interest information display areacrossover confirmation and priority determination unit configured todetermine interest information from the gaze of the first occupant andthe gaze of the second occupant, to determine whether areas fordisplaying the interest information on a windshield mutually crossoverwith each other, and to determine a priority of a display; and a displaycontrol unit configured to control the display in accordance with thepriority of the display.
 2. The system of claim 1, wherein the displaycontrol unit is configured to control the display in consideration ofpreset initiative information when a display area for the interestinformation of the first occupant and a display area for the interestinformation of the second occupant overlap with each other.
 3. Thesystem of claim 2, wherein the display control unit is configured tocontrol the display so that the display area for the interestinformation of the first occupant is displayed on a top position whilecovering the display area for the interest information of the secondoccupant, when the first occupant is set to have an initiative over thesecond occupant.
 4. The system of claim 2, wherein the display controlunit is configured to control the display by moving the display area forthe interest information of the second occupant so that the display areafor the interest information of the second occupant does not overlapwith the display area for the interest information of the firstoccupant, when the first occupant is set to have an initiative controlover the second occupant.
 5. The system of claim 2, wherein the displaycontrol unit is configured to control the display in consideration of animportance of the interest information, when the display area for theinterest information of the first occupant and the display area for theinterest information of the second occupant overlap with each other. 6.The system of claim 5, wherein the display control unit is configured tocontrol the display to display the interest information of the firstoccupant, without displaying the interest information of the secondoccupant, when it is determined that the interest information of thefirst occupant has higher importance than the interest information ofthe second occupant.
 7. The system of claim 2, wherein the displaycontrol unit is configured to determine whether to display the interestinformation of the second occupant in the display area assigned to thefirst occupant, in consideration of at least one of types of contentdisplayed in the display area assigned to the first occupant and anoperation situation in relation to the display area assigned to thefirst occupant.
 8. The system of claim 1, further comprising: a firstoccupant's external interest information confirmation unit configured toconfirm external interest information of the first occupant from thegaze of the first occupant; a second occupant's external interestinformation confirmation unit configured to confirm external interestinformation of the second occupant from the gaze of the second occupant;and an external interest information display area confirmation unitconfigured to confirm a display area for the external interestinformation of the first occupant and a display area for the externalinterest information of the second occupant, wherein the interestinformation display area crossover confirmation and prioritydetermination unit is configured to determine the priority for display,in consideration of recognition influence on the display area for theexternal interest information of the first occupant by the display areafor the external interest information of the second occupant withrelatively low importance of information display.
 9. The system of claim8, wherein the display control unit is configured to perform displaycontrol to display only the external interest information of the firstoccupant without displaying the external interest information of thesecond occupant when it is determined that the recognition influenceexceeds a preset value.
 10. The system of claim 8, wherein the displaycontrol unit is configured to perform control to display outlineinformation in respect to an original position of an external interestobject of the second occupant.
 11. The system of claim 8, wherein thedisplay control unit is configured to restore the display area for theexternal interest information of the second occupant to an area beforemoving in consideration of the gaze of the first occupant.
 12. A methodfor controlling a vehicle display based on an occupant's gaze, themethod comprising: (a) recognizing gaze information of a plurality ofoccupants in a vehicle; (b) confirming interest information using thegaze information; (c) confirming whether display areas for the interestinformation displayed on a windshield, which is a transparent display,crossover with each other, and (d) performing display control when thedisplay areas for the interest information crossover with each other.13. The method of claim 12, wherein in (d), the display control isperformed in consideration of preset initiative control informationaccording to a result of confirming that a display area for interestinformation of a first occupant and a display area for interestinformation of a second occupant overlap with each other.
 14. The methodof claim 13, wherein in (d), when the first occupant is set to have aninitiative control than the second occupant, the display control isperformed so that the display area for the interest information of thefirst occupant is displayed at a top level while covering the displayarea for the interest information of the second occupant.
 15. The methodof claim 13, wherein in (d), when the first occupant is set to have aninitiative control than the second occupant, the display control isperformed so that the display area for the interest information of thesecond occupant is moved so as to not overlap with the display area forthe interest information of the first occupant.
 16. The method of claim12, wherein in (d), when the display area for the interest informationof the first occupant and the display area for the interest informationof the second occupant overlap with each other, the display control isperformed in consideration of importance of the interest information.17. The method of claim 16, wherein in (d), when it is determined thatthe interest information of the first occupant has higher importancethan the interest information of the second occupant, the displaycontrol is performed so that the interest information of the secondoccupant is not displayed and the interest information of the firstoccupant is displayed.
 18. The method of claim 12, wherein in (d), it isdetermined whether the interest information of the second occupant isdisplayed in a display area assigned to the first occupant, inconsideration of at least any one of types of content displayed in thedisplay area assigned to the first occupant and an operation situationassociated with the display area assigned to the first occupant.